1. Field of the Invention
The present invention relates to a multiplexing apparatus and method, a transmitting apparatus and method, and a recording medium. More particularly, the present invention relates to a multiplexing apparatus and method, a transmitting apparatus and method, and a recording medium suitable for use, for example, in a situation where a bit stream of a digital signal formed from an image (moving image) signal and an audio signal is recorded on a recording medium such as a magneto-optical disk or a magnetic tape and is reproduced to be displayed on a display, or in a situation where such a bit stream is transmitted from a transmitting terminal to a receiving terminal over a transmission channel in a television conference system, a video telephone system, broadcasting apparatuses or the like, and is received at the receiving terminal to be displayed.
2. Description of the Related Art
FIG. 7 shows the configuration of an example of a conventional multiplexing/demultiplexing apparatus.
At a transmitting terminal, a digital image signal (video data) is supplied to a video encoder 101, which compresses and encodes a bit stream of video data in accordance with a standard, e.g., ISO13818-2 (so-called MPEG (Moving Picture Experts Group) 2 video standard), ISO11172-2 (so-called MPEG1 video standard) or the like, and thereafter outputs the compressed and encoded data. A digital audio signal (audio data) is supplied to an audio encoder 102, which compresses and encodes a bit stream of audio data in accordance with a standard, e.g., ISO13818-3 (so-called MPEG2 audio standard), ISO11172-3 (so-called MPEG1 audio standard) or the like, and thereafter outputs the compressed and encoded data. Another encoder 103 is supplied with digital data (other data) other than video data and audio data. The encoder 103 encodes and outputs the other data.
A bit stream compressed and encoded and thereafter output will be referred to as xe2x80x9celementary streamxe2x80x9d as occasion demands.
Elementary streams of video data, audio data and other data respectively output from the video encoder 101, the audio encoder 102 and the other encoder 103 are supplied to a multiplexer 104. The multiplexer 104 multiplexes the elementary streams of video data, audio data, other data, and additional data (additional information) necessary in a time division multiplexing manner at a receiving terminal. A stream obtained by this multiplexing (hereinafter referred to as xe2x80x9cmultiplexed streamxe2x80x9d as occasion demands) is recorded on a predetermined recording medium 105 or is transmitted through a predetermined transmission medium 106.
At the receiving terminal, the multiplexed stream reproduced from the recording medium 105 or received through the transmission medium 106 is supplied to a demultiplexer 107. The demultiplexer 107 separates the multiplexed stream into elementary streams of video data, audio data and other data, and additional data by referring to the additional data in the multiplexed stream. The elementary streams of video data, audio data and other data are respectively supplied to a video decoder 108, an audio decoder 109 and another decoder 110 to be respectively decoded into video data, audio data and other data in certain formats.
The decoded video data is supplied to, for example, a display unit such as a cathode ray tube (CRT) or a liquid crystal display to be displayed. The decoded audio data is supplied to, for example, a speaker or the like to be output. The other data is supplied to, for example, a predetermined recording medium to be recorded.
With respect to the above-described multiplexing of elementary streams at a transmitting terminal, a standard model (system target decoder (STD)) corresponding to a decoder at a receiving terminal is supposed and time division multiplexing is performed so that decoding (decode processing) can be correctly performed in the STD.
FIG. 8 shows the configuration of an example of an STD in accordance with ISO13818-1 (so-called MPEG2 system) and ISO11172-1 (so-called MPEG1 system).
A multiplexed stream formed by time-division-multiplexing elementary streams is input to the STD. The multiplexed stream contains time information (clock reference) as additional data, and is input to the STD at a time corresponding to the time information. The multiplexed stream also contains multiplexing rate information MUX_rate as additional data, and is input to the STD at a data rate corresponding to this multiplexing rate information.
The multiplexed stream input to the STD is supplied to a switch 111. The switch 111 selects one of input terminals of a video buffer 112, audio buffers 1131 to 113N and another buffer 114 to enable video data, audio data (N-channel audio data) or other data to be supplied to and stored in the corresponding buffer (video buffer 112, audio buffers 1131 to 113N) or buffer 114).
Each of the elementary streams of video data, audio data (N-channel audio data) and other data respectively stored in the video buffer 112, the audio buffers 1131 to 113N and the other buffer 114 is instantly read out by a length corresponding to its predetermined decoded unit.
Such a decoded unit is called an access unit. With respect to video data, one frame, for example, is prescribed as an access unit. With respect to audio data, one audio frame, for example, is prescribed as an access unit. The length of one audio frame of audio data varies depending upon the sampling frequency at which the audio data was sampled.
The multiplexed stream also contains the decoding time of each access unit as additional data. Each of access units of video data, audio data and other data stored in the video buffer 112, the audio buffers 1131 to 113N and the other buffer 114 is instantly read out at the corresponding decoding time to be supplied to a decoder 115, decoders 1161 to 116N, or a decoder 117. Video data decoded by the decoder 115 is output to a CRT display unit or the like via a reorder buffer 118 and a switch 119.
Each access unit is read out according to the decoding time to ensure synchronization between the elementary streams. That is, for example, synchronization between images and corresponding sounds is ensured.
Each of access units into which the data is separated by the switch 111 is stored till a predetermined time (the above-mentioned decoding time) in the video buffer 112, the audio buffers 1131 to 113N or the other buffer 114 of the STD. These buffers will be referred to collectively as xe2x80x9cSTD bufferxe2x80x9d. Accordingly, on the transmitting side, it is necessary to perform scheduling of time division multiplexing in order to prevent an overflow in the STD buffer (an excess of data accumulated in the STD buffer over the capacity of the STD buffer) or an underflow (failure to store one complete access unit in the STD buffer before the decoding time).
FIGS. 9A and 9B show the formats of multiplexed streams prescribed in ISO13818-1, for example.
A multiplexed stream is obtained by time-division-multiplexing at least one elementary stream and is formed of a plurality of packets. That is, each of elementary streams is separated into packets, and the time of being input to the decoder (clock reference), the decoding time at which each access unit is decoded (displayed), etc., are added to the packets, thus forming a multiplexed stream.
In ISO13818-1, as shown in FIGS. 9A and 9B, a transport stream (FIG. 9A) and a program stream (FIG. 9B) are prescribed. A transport stream is a stream for use in transmission in an environment having such an influence as to cause errors such as a bit error and cell loss. For example, such a stream is used in transmission channels of asynchronous transfer mode (ATM) networks and digital broadcasting. A program stream is a stream for use in transmission in an environment with a reduced possibility of occurrence of errors. In particular, such a stream is used with a recording medium in the form of a disk or a tape.
As shown in FIG. 9A, a transport stream is formed of transport stream packets of a fixed length in which elementary streams (of video data, audio data, and other data), etc., are set, and other packets in which are set an adaptation field containing a clock reference, etc., and additional information (additional information in a narrow sense) including, if the transport stream is a multi-program transport stream, program specific information (PSI) on selection from a plurality of programs, extraction of the desired packet and the method of decoding.
Each packet has a transport stream header, and the packets in which elementary streams are set may have packet headers, each packet header containing a decoding time, information on the contents of the packet, etc.
On the other hand, as shown in FIG. 9B, a program stream is formed of a pack in which a certain number of packets of a variable length are combined. A packet header is attached to each packet. With respect to packets in which elementary streams are set, a decoding time is set in the corresponding packet header.
Also, a packet header is attached to each pack. In the program stream, a clock reference is set in the pack header.
The program stream further contains a packet in which additional data (additional information in a narrow sense) such as a program stream directory (PSD) or a program stream map (PSM) is set.
The aforementioned additional data (additional information in a broad sense) represents data other than elementary streams in the multiplexed stream.
Time division multiplexing for obtaining the above-described multiplexed stream is performed based on two processings described below.
That is, it is necessary to perform, as the first processing, scheduling which comprises forming suitable additional data and determining data to be multiplexed and the sizes of the multiplexed data in order to prevent an overflow and an underflow in an STD buffer of an STD supposed to exist at a receiving terminal and in order to ensure synchronization between elementary streams. (This kind of processing will be referred to as xe2x80x9cmultiplexing scheduling processingxe2x80x9d as occasion demands.)
Then, it is necessary to perform, as the second processing, multiplexing of the elementary streams and the additional data on the basis of information on the data and the sizes determined by the multiplexing scheduling (hereinafter referred to as xe2x80x9cschedule informationxe2x80x9d as occasion demands). (This kind of processing will be referred to as xe2x80x9cbit stream multiplexing processingxe2x80x9d as occasion demands.)
FIG. 10 shows the configuration of an example of a multiplexing apparatus which performs the above-described multiplexing scheduling processing and bit stream multiplexing processing in a time division manner.
In this multiplexing apparatus, a scheduler which performs multiplexing scheduling processing and a bit stream multiplexer which performs bit stream multiplexing processing are not definitely separated from each other, and these two processings are performed by a scheduler and bit stream multiplexer 122.
That is, encoders 1211 to 121N encode video data and audio data into elementary streams, which are supplied to the scheduler and bit stream multiplexer 122. The scheduler and bit stream multiplexer 122 alternately performs multiplexing scheduling processing and bit stream multiplexing processing by using the elementary streams input thereto and by setting a schedule unit corresponding to, for example, a packet unit.
The problem of alternately performing multiplexing scheduling processing and bit stream multiplexing processing as described above, i.e., in a time division manner is that, if one of the two processings is delayed for some reason, the other processing is also delayed.
For example, a situation such as that shown in FIG. 11A may be considered where, when multiplexing scheduling processing S and bit stream multiplexing processing M are being performed alternately, a wait of outputting of the multiplexed stream occurs at a certain time and bit stream multiplexing processing M(n+1) of the (n+1)th packet is thereby delayed. In such a situation, multiplexing scheduling processing S(n+2) of the next (n+2)th packet cannot be performed before bit stream multiplexing processing M(n+1) is finished. Thus, a delay of bit stream multiplexing processing directly influences the next multiplexing scheduling processing.
Another situation shown in FIG. 11B, for example, may be considered where multiplexing scheduling processing S(n+1 of the (n+1)th packet is interrupted to be delayed. Also in such a situation, multiplexing scheduling processing M(n+1) of the (n+1)th packet cannot be performed.
As described above, a delay of one of the two processings directly influences the other processing, so that processing delay times are successively accumulated. When the length of the accumulated delay time is increased to a certain value, it becomes difficult to perform time division multiplexing in a real time required for, for example, broadcasting.
As a method for avoiding this problem, there is a method of reducing the multiplexing rate. However, if such a method is used, a high bit rate cannot be achieved. There is also a method of using an apparatus capable of performing multiplexing scheduling processing and bit stream multiplexing processing at a high speed. The entire apparatus for the processing of this method, however, is disadvantageously expensive. Further, there is a method of providing a buffer having a large capacity in a stage following the scheduler and bit stream multiplexer 122, sufficiently accumulating the multiplexed stream in the buffer and thereafter outputting the multiplexed stream. The entire apparatus for the processing of this method, however, is disadvantageously expensive and large in size.
In a case where images formed by a video camera, for example, are multiplexed by the above-described multiplexing apparatus and recorded by a recording apparatus, if a waiting time of the recording apparatus occurs, it propagates as a processing delay from bit stream multiplexing processing to multiplexing scheduling processing. In such a situation, processing of the images successively output from the video camera cannot be performed suitably. To prevent such waiting time propagation, a large-capacity buffer is also required, result in increases in the cost and size of the entire apparatus.
FIG. 12 shows the configuration of an example of a multiplexing apparatus which performs multiplexing scheduling processing and bit stream multiplexing processing separately from each other.
The sizes and decoding (display) times of access units constituting elementary streams output from encoders 1211 to 121N are supplied as access unit information to a scheduler 131, in which data to be next multiplexed and the size of the data, i.e., schedule information, are calculated on the basis of access information.
While, in the arrangement shown in FIG. 12, access unit information is obtained from the encoders 1211 to 121N outputting elementary streams, processing for extracting access unit information from elementary streams may be performed as a preprocessing before multiplexing scheduling processing if the encoders 1211 to 121N cannot output access unit information.
It is not necessary to limit the information output as access unit information by the encoders 1211 to 121N to the sizes and decoding times of access units. Information from which the sizes and decoding times of access units can be obtained may suffice. For example, with respect to image data encoded in accordance with the ISO13818-2 standard, relative values of decoding times can be obtained if the frame rates of elementary streams of the image data and repeat_first_field/picture_coding_type of access units are known.
The scheduler 131 sets, for example, a packet unit or the like as one schedule unit, calculates schedule information and additional data on the basis of access information, and commands (outputs a multiplexing command to) a bit stream multiplexer 132 to perform multiplexing by outputting these information and data to the bit stream multiplexer 132.
The bit stream multiplexer 132 is supplied with elementary streams from the encoders 1211 to 121N, and time-division-multiplexes the elementary streams and additional data from the scheduler 131 by a multiplexing command from the scheduler 131 and in accordance with schedule information from the scheduler 131.
After finishing the bit stream multiplexing processing, the bit stream multiplexer 132 outputs a multiplexing end notice to the scheduler 131 to inform the same of the end of the processing, and waits for the next multiplexing command from the scheduler 131. On the other hand, when the scheduler 131 receives a multiplexing end notice, it outputs a multiplexing command together with the next schedule information and additional data if the calculation of the next schedule information and additional data has been completed.
Successively, the scheduler 131 and the bit stream multiplexer 132 repeat the same processings.
As described above, the scheduler 131 and the bit stream multiplexer 132 respectively perform multiplexing scheduling processing and bit stream multiplexing processing separately from each other. In such a case, while the bit stream multiplexer 132 is performing bit stream multiplexing processing, the scheduler 131 can start the next multiplexing scheduling processing. Thus, the influence of a delay of bit stream multiplexing processing upon the scheduler 131 is reduced in comparison with the case shown in FIG. 10. It is also possible to speed up the processing as a whole.
However, the scheduler 131 cannot transmit the next schedule information and additional data to the bit stream multiplexer 132 unless it receives a multiplexing end notice from the bit stream multiplexer 132. Therefore, if, for example, as shown in FIG. 13A, a long wait of outputting of the multiplexed bit stream by the bit stream multiplexer 132 occurs, the delay of the outputting propagates to multiplexing scheduling processing of the scheduler 131. That is, in such a case, the scheduler 131 cannot start the next multiplexing scheduling processing before it receives a multiplexing end notice. FIG. 13A shows a state in which bit stream multiplexing processing M(n+1) of the (n+1)th packet is delayed and multiplexing scheduling processing S(n+3) of the (n+3)th packet cannot be started.
If, for example, as shown in FIG. 13B, multiplexing scheduling processing is delayed for a long time by an interrupt in multiplexing scheduling processing, bit stream multiplexing processing cannot be started, also resulting in a delay of the processing. FIG. 13B shows a state in which multiplexing scheduling processing S(n+1) of the (n+1)th packet is delayed and bit stream multiplexing processing M(n+1) of the (n+1)th packet cannot be started.
That is, in the multiplexing apparatus shown in FIG. 12, since multiplexing scheduling processing is performed with a one-cycle precedence, a delay corresponding to one processing cycle can be absorbed. However, there is no means for coping with a longer delay, and there is the same problem as that relating to FIG. 10.
In view of the above-described circumstances, an object of the present invention is to provide a multiplexing apparatus and method, a transmitting apparatus and method, and a recording medium arranged to minimize the propagation of a delay of one of multiplexing scheduling processing and bit stream multiplexing processing to the other of these processings.
To achieve the above-described object, according to one aspect of the present invention, there is provided a multiplexing apparatus which time-division-multiplexes at least one digital signal bit stream and additional information added to the bit stream, the apparatus comprising schedule information calculation means for calculating schedule information on a schedule of time division multiplexing on the basis of access unit information on an access unit prescribed in the bit stream, schedule information storage means for storing the schedule information output by the schedule information calculation means, additional information calculation means for calculating the additional information, additional information storage means for storing the additional information output by the additional information calculation means, and multiplexing means for time-division-multiplexing the bit stream and the additional information on the basis of the schedule information stored by the schedule information storage means.
According to another aspect of the present invention, there is provided a multiplexing method of time-division-multiplexing at least one digital signal bit stream and additional information added to the bit stream, the method comprising the steps of calculating schedule information on a schedule of time division multiplexing on the basis of access unit information on an access unit prescribed in the bit stream, and calculating the additional information, storing the schedule information and the additional information, and time-division-multiplexing the bit stream and the additional information on the basis of the schedule information.
According to still another aspect of the present invention, there are provided a transmitting apparatus, a transmitting method and a recording medium arranged to transmit or record a multiplexed stream formed by time-division-multiplexing at least one digital signal bit stream and additional information be added to the bit stream. The multiplexed stream transmitted by the transmitted by these apparatus and method or recorded on the recording medium is obtained by calculating schedule information on a schedule of time division multiplexing on the basis of access unit information on an access unit prescribed in the bit stream, by calculating the additional information, by storing the schedule information and the additional information, and by time-division-multiplexing the bit stream and the additional information on the basis of the schedule information.
In the above-described multiplexing apparatus, the schedule information calculation means calculates schedule information on a schedule of time division multiplexing on the basis of access unit information on an access unit prescribed in a bit stream, and the schedule information storage means stores the schedule information. The additional information calculation means calculates additional information, and the additional information storage means stores the additional information. The multiplexing means time-division-multiplexes the bit stream and the additional information on the basis of the schedule information stored by the schedule information storage means.
In the above-described multiplexing method, schedule information on a schedule of time division multiplexing is calculated on the basis of access unit information on an access unit prescribed in a bit stream, additional information is calculated, the schedule information and the additional information are stored, and the bit stream and the additional information are time-division-multiplexed on the basis of the schedule information.
In the above-described transmitting apparatus, transmitting method, the recording medium, a multiplexed stream is obtained by calculating schedule information on a schedule of time division multiplexing on the basis of access unit information on an access unit prescribed in a bit stream, by calculating additional information, by storing the schedule information and the additional information, and by time-division-multiplexing the bit stream and the additional information on the basis of the schedule information.